3. Results: Whole Brain (316 ROIs)¶
3.1. Search Grid With 5-fold Cross Validation¶
The model was fine tuned by finding optimal values for the following hyperparmeters:1) L2 regularization and 2) dropout rate applied to each of the hidden layers, and 3) learning_rate of the Adam optimizer.
Optimal hyperparameters were found by doing a full grid-search over 75 combinations of hyperparameteres, and cross-validating each combination with a nested 5-fold cross-validation method. A total of 75 models were trained and validated. Following plot shows the performance of each model in terms of its training and validation accuracies. The models are shown in the descending order of their mean validation accuracy. The error bars indicate standard deviation across folds.
Best performing model yielded mean training and validation accuracies of 0.89 and 0.82, respectively. Its hyperparameters were: 1) L2 = 0.003, dropout = 0.3, and learning_rate = 0.001.
3.2. Test Data Performance¶
3.2.1. Accuracy¶
The trained model was tested on the near-miss segments of the 19 held-out participants. Following figure shows temporal and overall accuracies on the held-out participants. The model performs resonably well from the 1st timepoint (TP) itself, with a mean accuracy of 0.8. The mean accuracy steadily increases to 0.89 by the 7th TP. “Overall” accuracy is the mean accuracy across TP, which is 0.83.
3.2.2. Probability of predicting the true class as a function of time¶
Following figure shows probability of predicting the true class as a function of time. The probability of predicting the true class increases with time.
3.3. Chance Accuracy¶
To assess significance of the model performance, the observed test accuracy was compared against the accuracy that would be observed if the model was to guess one of the two classes at random. To that end, model with the best hyperparameter settings was trained on the training set a hundred times, each time with randomly shuffled labels. At every iteration, the model was tested on the test set with “non-shuffled” (i.e., true) labels. This process was meant to simulate a chance accuracy distribution. The mean of the chance accuracy distribution formed the baseline performance measure against the observed performance of the model when trained on true labels. The observed test accuracy was significantly greater than the average chance accuracy (p < 0.009). See the figure below.
Accuracy
Observed: 0.83
Chance: 0.50
Observed > Chance (p = 0.0099)
3.4. Temporal Trajectories¶
GRU outputs hidden states that are typically high dimensional. Hidden states (\(h_{t}\)) capture spatio-temporal variance that is most useful in maintaning class separability. To visualize dynamics, \(h_{t}\) was linearly projected onto a lower (3D) dimensional space, \(\hat{h_{t}}\). This was done by replacing the output layer with a Dimensionality Reduction Dense Layer (DRDL) with three linear units. In essence, this is a supervised non-linear dimensional reduction step.
The 3-dimensional representations of \(h_{t}\) (\(\hat{h_{t}}\)) for both stimulus class are plotted along the three axes of the coordinate system below. The plot represents the temporal trajectories of the two classes. At the first timepoint the two classes are closest to each other. Distance between them increases with every timepoint. Next plot shows the Euclidean distance between the two classes as a function of time.
The above trajectories and Euclidean distance plots suggest that the GRU architecture is able to characterize distinct spatio-temporal patterns in the fMRI data, for appraoching and retreating threats.
3.5. Demonstration¶
The following video clip demonstrates model performance on one of the held out participant’s fMRI data. The video shows the visual paradigm presented to the participant during his/her fMRI scan, along with model predictions at the top-right of the screen. Prediction is either “Approach” or “Retreat”. If the prediction is correct, the color of the text remains green; and if it is incorrect, it turns red. Note that when the circles touch, the screen turns white and a red wheel appears around the circles to indicate delivery of the physical shock. Also note that the speed of the video has been increased by 4x for quick demonstration purposes.